Nice job. NPU support is huge for this project. Do you think its also possible to make it work on Exynos 2200 and 2400 NPUs?

Nice job. NPU support is huge for this project. Do you think its also possible to make it work on Exynos 2200 and 2400 NPUs?

thanks for your kind comment.

1. Quacomm's Hexagon NPU support is really huge work for this project although now we clearly know the principle or know what, because Qualcomm provides some binary dedicated tools to do LLM model conversion in their dedicated AI sw stacks and some other closed-source implementation also use this similar approach exactly. so programmers must compose an ideal QNN graph according to the complete ggml cgraph manually in ggml-qnn backend if they use/chose the second tech approach in ggml-qnn backend("mapping the complete ggml cgraph to a single opcfg QNN graph"). there are 800+ cgraph nodes and 50+ ops in qwen1_5-1_8b-chat-q4_0.gguf, accordingly, "(Hexgon) NPU support is huge for this project", real AI experts must be involved in the rest parts of ggml-qnn.
2. I think I can make it(ggml-exynos or ggml-samsung) work on Exynos 2200 if I can get a necessary phone(I can try to buy it) and SDK&tech docs(this might-be not easy because of strict IPR policy in some big IT companys as my personal understanding at the moment) and follow the principle "make it run and then make it right and finally make it fast",this is one of my areas of expertise.

Nice work!

However, will offloading MatMul to CDSP layer by layer incur a significant FastRPC overhead?

As far as I know, with zero-copy enabled, the overhead of a single FastRPC call is around 1–3ms (on the 8750 device).

Could you please share the end-to-end performance results?

Nice work!

However, will offloading MatMul to CDSP layer by layer incur a significant FastRPC overhead?

1.can be avoid by the same mechanism in QNN SDK: shared-buffer or memory pool between AP and cDSP through ION memory or DMA memory.
2.a senior staff tech expert from Qualcomm's headquarter has told me: QNN is not the right solution here.
3.various local experiments has confirmed that QNN-CPU&QNN-NPU's performance is really bad then the default ggml cpu backend, and also really bad then general approach through cDSP directly.

As far as I know, with zero-copy enabled, the overhead of a single FastRPC call is around 1–3ms (on the 8750 device).

Could you please share the end-to-end performance results?

what's the means of "end-to-end performance results"?

1. this link opencl: improve profiling #12442 (comment) might be useful for your question.

2. verify the "end-to-end performance" manually.

git clone https://github.com/kantv-ai/ggml-hexagon
cd ggml-hexagon
git checkout pr_to_upstream
./scripts/build-run-android.sh build
./scripts/build-run-android.sh run_llamacli

modify the inference_approach manually for verify approach through QNN or cDSP:

zhouwg:$ more scripts/ggml-qnn.cfg 
[general]
#0: QNN-CPU backend
#1: QNN-GPU backend
#2: QNN-NPU backend
#3: default ggml backend
qnn_backend = 2

# enable/disable QNN's internal log
print_qnn_internal_log = 0

# enable/disable perf of op function
enable_perf = 1

# enable/disable print tensors info in op function
print_tensors_info = 0

# enable/disable dump op info in handle_op
dump_op_info = 0

# 0: general approach through QNN
# 1: general approach through Hexagon cDSP
# 2: special approach through QNN: mapping entire ggml cgraph to QNN graph(beyond scope of this PR)
inference_approach = 1

[npu]
hvx_threads = 4
vtcm_size_in_mb = 8
enable_dlbc = 1
precision_mode = "fp16"

Thanks for your feedback!

Nice work!
However, will offloading MatMul to CDSP layer by layer incur a significant FastRPC overhead?

1.can be avoid by the same mechanism in QNN SDK: shared-buffer or memory pool between AP and cDSP through ION memory or DMA memory 2.a senior staff tech expert from Qualcomm's headquarter has told me: QNN is not the right solution here. 3.various local experiments has confirmed that QNN-CPU&QNN-NPU's performance is really bad then the default ggml cpu backend.

I understand using techs like shared-buffer will reduce fastrpc overhead (as I mentioned "with zero-copy enabled"). However, LLM inference requires offloading hundreds of layers. Even with shared-buffer, this overhead remains significant.So, what I’d like to ask is whether you have any other optimization strategies to address this issue.

I see that you have actually analyzed and found it infeasible to transfer the entire graph to QNN. However, if offloading is done on a per-layer basis, it seems that the overhead of these fastRPCs can never be optimized.

As far as I know, with zero-copy enabled, the overhead of a single FastRPC call is around 1–3ms (on the 8750 device).
Could you please share the end-to-end performance results?

what's the means of "end-to-end performance results"?

I'm sorry for this confusing question. End to End performance results means the prefill & decode speed (tokens/s). I see your comments mentioned some results, but these results seems incomplete.

git clone https://github.com/kantv-ai/ggml-hexagon
cd ggml-hexagon
git checkout pr_to_upstream
./scripts/build-run-android.sh build
./scripts/build-run-android.sh run_llamacli

modify the inference_approach manually for verify approach through QNN or cDSP:

zhouwg:$ more scripts/ggml-qnn.cfg 
[general]
#0: QNN-CPU backend
#1: QNN-GPU backend
#2: QNN-NPU backend
#3: default ggml backend
qnn_backend = 2

# enable/disable QNN's internal log
print_qnn_internal_log = 0

# enable/disable perf of op function
enable_perf = 1

# enable/disable print tensors info in op function
print_tensors_info = 0

# enable/disable dump op info in handle_op
dump_op_info = 0

# 0: general approach through QNN
# 1: general approach through Hexagon cDSP
# 2: special approach through QNN: mapping entire ggml cgraph to QNN graph(beyond scope of this PR)
inference_approach = 1

[npu]
hvx_threads = 4
vtcm_size_in_mb = 8
enable_dlbc = 1
precision_mode = "fp16"

Thank you for providing these running commands, I'll try it.

Additionally, I would like to provide an extra piece of opinion. The reason why performance on Qualcomm's official AI hub is good enough. I think it's because they convert the entire graph to QNN, which gives them three exclusive advantages in hardware utilization:

1. The overhead of fastrpc is small;
2. They can take advantage of HMX instructions and VTCM high-speed memory;
3. Comprehensive graph optimization.

By the way, I took a quick look at the operator code you implemented on cdsp and it seems that you haven't used HVX intrinsics yet. This might be one of the reasons for the suboptimal performance.

I suggest you take a look at the linear algebra library provided by Qualcomm for cdsp. Although their performance is not great either.

Additionally, I would like to provide an extra piece of opinion. The reason why performance on Qualcomm's official AI hub is good enough. I think it's because they convert the entire graph to QNN, which gives them three exclusive advantages in hardware utilization:

1. The overhead of fastrpc is small;

the overhead of FastRPC should be same in various tech approaches, I personally think the overhead through cDSP directly might-be minimum:

datapath through QNN:

user code(ggml-hexagon backend)  <------> QNN API <------> QNN SDK <------> FastRPC framework(user-space lib and kernel driver) in HLOS(here is Android OS) <------> embedded OS on cDSP <------> FastRPC framework on cDSP <------> Hexagon nn libs on cDSP

datapath through cDSP directly:

user code(ggml-hexagon backend, similar to  TEE CA) <------> FastRPC framework(user-space lib and kernel driver) in HLOS(here is Android OS) <------> embedded OS on cDSP <------> FastRPC framework on cDSP <------> user code on cDSP(hexagon kernels, similar to TEE TA, or opencl kernels in ggml-opencl, or cuda kernels in other backends)

I think that's why the senior staff tech expert from Qualcomm headquarter said "QNN is not the right solution here", in the fact, the NPU performance through QNN is really bad here(ggml / llama.cpp).

because we can't utilize the dedicated binary tools which provided by Qualcomm here, or we(llama.cpp community) can't re-create an entire Qualcomm's dedicated AI stack in ggml/llama.cpp(mapping the all ggml ops and entire ggml cgraph to a single QNN graph is also not easy to Qualcomm's world-class engineering team, I'll prefer to do some adaptation efforts with Intel's sycl stack if I'm a regular employee of Qualcomm's AI team, that's a more practical and desire direction). accordingly, offload some performance-sensitive ggml ops to cDSP directly is a practical way/direction: we only need to focus on hexagon kernels through various highly-designed algorithms or HVX instructions on cDSP, this is also the key-reason why I think this PR should be approved(we don't need to do more complex things in ggml-qnn.cpp from now on).

at the same time, we can clearly see that the so-called FastRPC mechanism or framework is exactly similar to mechanism in TEE.

2. They can take advantage of HMX instructions and VTCM high-speed memory;

because QNN SDK's internal will indirectly calling Qualcomm's Hexagon nn libs on cDSP which might-be/should-be highly-optimized with HVX SIMD instructions(they have plenty of excellent software engineers or AI experts).

3. Comprehensive graph optimization.

I agree with your opinion: QNN's internal did some things with the specified single QNN graph you pointed out. pls refer to section "Big picture of ggml-hexagon backend" in this PR, or refer to:jeffzhou2000#24

Can Hexagon SDK not be used in termux environment?
It doesn't seem to be provided in a zip format.
I'm in trouble because I only have an android.

Can Hexagon SDK not be used in termux environment?

Hexagon SDK need to be used in a standard Linux environment(Ubuntu 20.04/22.04 is recommended) to build the entire ggml-hexagon source code in this PR, pls refer to section "How to build ggml‐hexagon source code for Android and verify ggml-hexagon backend on Snapdragon based phone" in PR description.

btw, I'm not familiar with termux environment but I think that's a limited Linux running environment on Android device. in the fact, we can verify/validate this PR through the self-made script build-run-android.sh on a standard Linux OS(or through Linux VM on Window10/11, or through WSL2 on Windows10/11) easily and directly, so termux might-be not needed or suitable here(I'm not sure about this because I didn't verify this opinion).

It doesn't seem to be provided in a zip format. I'm in trouble because I only have an android.

currently, it need Qualcomm account to get the Hexagon SDK to build the entire ggml-hexagon source code, this is the key-reason why build-run-android.sh can't download Hexagon SDK automatically(the self-made script build-run-android.sh can download Android NDK and QNN SDK automatically to make workflow easily).

at the same time, I cannot share the local self-made zip format Hexagon SDK publicly or I cannot distribute the Hexagon SDK in another way, because the Hexagon SDK must be obtained with a Qualcomm Developer Account and the self-made local zip format Hexagon SDK contains/binds my personal Qualcomm account&license information.

this IPR policy is make sense(my personal opinion) because Qualcomm already(we developers and programmers must thanks to @slaren) public it's unlimited QNN SDK freely and it is clear that the Hexagon SDK is more valuable although the Hexagon SDK is also free to developers and Qualcomm need to know how many developers use their unique and valuable Hexagon SDK. of course, it will make developer's workflow more easily if Qualcomm can also public an unlimited Hexagon SDK because I also think "more people/companies use the world-class Snapdragon mobile/vehicle/desktop SoC is another key-point".

@zhouwg

Thank you for your answer. It seems that the only way is to unzip it using googlecolab.

@zhuwg

I would like to specify the termux lib path, but it is fixed and I would like to change it.

ggml-hexagon.cpp

        .runtimelib_path        = "/data/local/tmp/",

QNN_DEFAULT_LIB_SEARCH_PATH

@zhuwg

I would like to specify the termux lib path, but it is fixed and I would like to change it.

ggml-hexagon.cpp

        .runtimelib_path        = "/data/local/tmp/",

QNN_DEFAULT_LIB_SEARCH_PATH

thanks for your comment,
might-be we can add a runtime configure item in scripts/ggml-hexagon.cfg, then user can dynamically adjust runtime lib path accordingly.

what's your suggestion for this problem? thanks.

@zhuwg

I would like to specify the termux lib path, but it is fixed and I would like to change it.

ggml-hexagon.cpp

        .runtimelib_path        = "/data/local/tmp/",

QNN_DEFAULT_LIB_SEARCH_PATH

thanks for your comment,
might-be we can add a runtime configure item in scripts/ggml-hexagon.cfg.

what's your suggestion for this problem? thanks.

Is it not enough to just replace it with QNN_DEFAULT_LIB_SEARCH_PATH?

When I copied what I built on GoogleColab to Termux and ran it, I got the following error.

device: GT5 pro(snapdragon gen3)

[bin]$ ls $PREFIX/lib/libQnn
libQnnCpu.so
libQnnGpu.so
libQnnHtp.so
libQnnHtpNetRunExtensions.so
libQnnHtpOptraceProfilingReader.so
libQnnHtpPrepare.so
libQnnHtpProfilingReader.so
libQnnHtpV68CalculatorStub.so
libQnnHtpV68Skel.so
libQnnHtpV68Stub.so
libQnnHtpV69CalculatorStub.so
libQnnHtpV69Skel.so
libQnnHtpV69Stub.so
libQnnHtpV73CalculatorStub.so
libQnnHtpV73Skel.so
libQnnHtpV73Stub.so
libQnnHtpV75CalculatorStub.so
libQnnHtpV75Skel.so
libQnnHtpV75Stub.so
libQnnHtpV79CalculatorStub.so
libQnnHtpV79Skel.so
libQnnHtpV79Stub.so
libQnnSystem.so
[bin]$ LD_LIBRARY_PATH=".:/vendor/lib64" ./llama-server
 
[ggmlhexagon_load_cfg, 1756]: load hexagon appcfg from 
/data/data/com.termux/files/usr/lib/ggml-hexagon.cfg
[operator(), 1762]: section[cdsp      ],[enable_rpc_dma
_mempool   ] = [0]
[operator(), 1762]: section[cdsp      ],[enable_rpc_ion
_mempool   ] = [0]
[operator(), 1762]: section[qnn       ],[precision_mode
           ] = ["fp16"]
[operator(), 1762]: section[qnn       ],[enable_dlbc   
           ] = [1]
[operator(), 1762]: section[qnn       ],[vtcm_size_in_m
b          ] = [8]
[operator(), 1762]: section[qnn       ],[hvx_threads   
           ] = [4]
[operator(), 1762]: section[general   ],[hwaccel_approa
ch         ] = [2]
[operator(), 1762]: section[general   ],[print_qnn_inte
rnal_log   ] = [0]
[operator(), 1762]: section[general   ],[enable_q_mulma
t          ] = [0]
[operator(), 1762]: section[general   ],[print_tensors_
info       ] = [0]
[operator(), 1762]: section[general   ],[hexagon_backen
d          ] = [2]
[operator(), 1762]: section[general   ],[dump_op_info  
           ] = [0]
[operator(), 1762]: section[general   ],[enable_perf   
           ] = [1]
[operator(), 1762]: section[general   ],[version       
           ] = ["1.00"]
[ggmlhexagon_load_cfg, 1786]: internal ggml_hexagon_ver
sion=1.80
[ggmlhexagon_load_cfg, 1787]: internal ggml_dsp_version
=0.60
[ggmlhexagon_load_cfg, 1788]: external ggml_hexagon_ver
sion=1.00
[ggmlhexagon_load_cfg, 1790]: hwaccel_approach=2(HWACCE
L_CDSP)
[ggmlhexagon_load_cfg, 1792]: hexagon_backend=2(HEXAGON
_BACKEND_CDSP)
[ggmlhexagon_load_cfg, 1793]: runtime libpath=/data/dat
a/com.termux/files/usr/lib/
[ggmlhexagon_load_cfg, 1794]: enable_perf=1
[ggmlhexagon_load_cfg, 1795]: enable_profiler=0
[ggmlhexagon_init_dsp, 5209]: init Hexagon cDSP with ba
ckend 2(HEXAGON_BACKEND_CDSP)
[ggmlhexagon_init_dsp, 5280]: using Hexagon domain 3(He
xagon-cDSP)
[ggmlhexagon_init_dsp, 5281]: unsignedpd_enabled 1
[ggmlhexagon_init_dsp, 5327]: error 0x80000406: failed 
to open domain 3(Hexagon-cDSP)
[ggmlhexagon_deinit_cdsp, 5172]: enter ggmlhexagon_dein
it_cdsp
[ggmlhexagon_deinit_cdsp, 5185]: leave ggmlhexagon_dein
it_cdsp
[ggml_backend_hexagon_reg, 6376]: init hexagon dsp fail
ure
/root/.builder/source/bin/llama-cpp-hexagon-branch-pr_t
o_upstream/ggml/src/ggml-hexagon/ggml-hexagon.cpp:6378:
 GGML_ASSERT(0 == result) failed

@zhouwg
I would like to specify the termux lib path, but it is fixed and I would like to change it.
ggml-hexagon.cpp

        .runtimelib_path        = "/data/local/tmp/",

QNN_DEFAULT_LIB_SEARCH_PATH

thanks for your comment,
might-be we can add a runtime configure item in scripts/ggml-hexagon.cfg.
what's your suggestion for this problem? thanks.

Is it not enough to just replace it with QNN_DEFAULT_LIB_SEARCH_PATH?

env var might-be a good idea although it seems equivalent to a new runtime configuration item in scripts/ggml-hexagon.cfg. there are many runtime configuration items at the moment, an uniform approach might-be better for developers and users.